docs: cover example solutions and check design for scorers - #205
Open
czenko wants to merge 1 commit into
Open
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
czenko
marked this pull request as ready for review
August 14, 2026 19:28
Guidance from the review of #168, generalized past the RLS eval so it holds for all 44 scenarios. - Checking your scorer: one green solution, a few bad ones, and the expected failures written down before scoring. - Solution shapes: a workspace overlay, a remote-shaped seed, or a report, depending on what the eval measures. Eight evals score nothing but the report. - Designing checks: what a check does when the object it inspects is absent, and finding a class of object from the system that owns it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
czenko
force-pushed
the
mirandalimonczenko/docs-1324-document-example-solutions-and-check-design-in-the-evals
branch
from
August 14, 2026 19:30
3a9e5d1 to
66ed0da
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes DOCS-1324
Problem
Ten more docs evals are scoped. The RLS eval took several review rounds, and most of the findings were scorer defects a contributor can catch before asking for review.
CONTRIBUTING.mddoesn't say how. Guidance comes from the review of #168.Solution
remote/-shaped seed for tools evals, and a report for the 8 evals scored on what the agent said rather than what it built.solutions/in the eval folder contents, above the optional entries, and links the new section from Submitting evals for review.Manual testing
Checking your scorerlink in that page. Both jump to the new section.